Multifunctional visualization instrument with orientation control

ABSTRACT

A multifunctional laryngoscope is provided that includes a handle comprising a proximal end and a distal end and a display screen on the handle. The laryngoscope includes a laryngoscope camera at the distal end of the handle and connects to an introducer comprising an orientation sensor at a distal end of the introducer. The laryngoscope includes a processor programmed to execute instructions for receiving from a steering input a steering command in a first reference frame, and mapping the steering command to a second reference frame oriented to the distal end of the introducer based on an orientation signal from the orientation sensor.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority to and the benefit of U.S.Provisional Application No. 62/812,678 filed on Mar. 1, 2019, thedisclosure of which is incorporated by reference in its entirety for allpurposes.

BACKGROUND

The present disclosure relates generally to medical devices and, moreparticularly, to a method of controlling a steerable introducer, such asa flexible endoscope.

Introducers are long flexible instruments that can be introduced into acavity of a patient during a medical procedure, in a variety ofsituations. For example, one type of introducer is a flexible endoscopewith a camera at a distal end. The endoscope can be inserted into apatient's mouth or throat or other cavity to help visualize anatomicalstructures, or to help perform procedures such as biopsies or ablations.Another type of introducer is a blind bougie (with no camera) which maybe inserted and then used to guide another device (such as anendotracheal tube) into place. These and other introducers may include asteerable distal tip that can be actively controlled to bend or turn thedistal tip in a desired direction, to obtain a desired view or tonavigate through anatomy. However, these steerable introducers can bedifficult to maneuver into the desired location and orientation within apatient's anatomy.

SUMMARY

Certain aspects or embodiments commensurate in scope with the originallyclaimed subject matter are summarized below. These aspects orembodiments are not intended to limit the scope of the disclosure.Indeed, the present disclosure may encompass a variety of forms that maybe similar to or different from the aspects set forth below.

In one aspect or embodiment, there is provided a steerable introducersystem that includes a laryngoscope and an introducer. The laryngoscopeincludes a handle comprising a proximal end and a distal end, a displayscreen on the handle, and a laryngoscope camera at the distal end of thehandle. The laryngoscope also includes a steering input for steering anintroducer, the steering input located on the handle or the displayscreen. The introducer is coupled to the handle and has an orientationsensor at a distal end of the introducer. The laryngoscope also includesa processor within the laryngoscope programmed to execute instructionsfor receiving from the steering input a steering command in a firstreference frame, and mapping the steering command from the firstreference frame to a second reference frame oriented to the distal endof the introducer based on an orientation signal from the orientationsensor.

The processor may be further programmed to execute instructions forgenerating a control signal for steering the introducer according to themapped steering command. The second reference frame may be defined by anangular offset from the first reference frame. Mapping the steeringcommand to the second reference frame may comprise adjusting thesteering command by the angular offset. The first reference frame may bedefined by a user input. The first reference frame may be defined byautomatic image recognition. The processor may be programmed to receivean image from the laryngoscope camera to identify a feature of the imageto perform the automatic image recognition. The processor may beprogrammed to receive an image from an introducer camera at the distalend of the introducer to identify a feature of the image to perform theautomatic image recognition. The display screen may display an imagefrom the laryngoscope camera in the first reference frame. The displayscreen may display an image from an introducer camera at the distal endof the introducer in the first reference frame.

In a further aspect or embodiment, which may be provided independently,there is provided an endoscope controller that includes a handle, adisplay screen on the handle, an endoscope port located on the handle orthe display screen, and a user input located on the handle or thedisplay screen. A processor within the controller is programmed toexecute instructions for receiving from the user input a steeringcommand in a user reference frame, receiving, from an endoscope coupledto the endoscope port, an orientation signal from an orientation sensorat an endoscope distal end, and translating the steering command as afunction of the orientation signal.

The processor may be further programmed to execute instructions forsteering the endoscope according to the translated steering command. Thecontroller may further comprise the endoscope coupled to the endoscopeport, wherein the endoscope may comprise an orientation sensor thatgenerates the orientation signal.

In another aspect or embodiment, which may be provided independently, amethod for controlling a steerable introducer includes receiving, at aprocessor, an orientation signal from an orientation sensor located at adistal end of a steerable introducer. The orientation signal defines anangular orientation of the distal end of the introducer. The method alsoincludes receiving, at the processor, a steering command comprising asteering direction in a user reference frame, translating the steeringcommand from the user reference frame to the angular orientation of thedistal end of the introducer, and steering the distal end of theintroducer according to the translated steering command.

The user reference frame may be defined in reference to an anatomicalfeature of the patient. The user reference frame may be defined by auser input. The method may further comprise receiving, at the processor,an image from a camera at the distal end of the introducer; rotating theimage into the user reference frame; and displaying the rotated image ata display screen.

In another aspect or embodiment, which may be provided independently, amethod for controlling a steerable introducer includes receiving, at aprocessor, a steering command from a user input and an orientationsignal from an orientation sensor of a steerable introducer. The methodalso includes translating, at the processor, the steering command as afunction of the orientation signal, and steering the introduceraccording to the translated steering command.

In a further aspect or embodiment, which may be provided independently,a method for controlling a steerable introducer includes receiving, at aprocessor, a steering command from a user input and an orientation inputfrom an orientation sensor. The method also includes generating, at theprocessor, a variable steering signal comprising steering instructionsthat vary as a function of both the steering command and the orientationinput, and steering the introducer according to the variable steeringsignal.

In another aspect or embodiment, which may be provided independently, amethod includes receiving, at a processor, a laryngoscope image from alaryngoscope camera; receiving, at the processor, an endoscope imagefrom an endoscope camera at a distal end of an endoscope and anorientation signal from an orientation sensor at the distal end of theendoscope; receiving a user input to establish a reference frame of thedistal end; receiving an updated signal from the orientation sensor thatindicates that the distal end has rotated away from the reference frame;and rotating an updated endoscope image into the reference frame basedon the updated signal.

Features in one aspect or embodiment may be applied as features in anyother aspect or embodiment, in any appropriate combination. For example,any one of system, laryngoscope, controller, introducer, or methodfeatures may be applied as any one or more other of system,laryngoscope, controller, introducer, or method features.

BRIEF DESCRIPTION OF THE DRAWINGS

Advantages of the disclosed techniques may become apparent upon readingthe following detailed description and upon reference to the drawings inwhich:

FIG. 1 is a perspective view of a multifunctional controller andsteerable introducer of a steerable introducer system, in accordancewith certain embodiments of the disclosure.

FIG. 2 is a perspective view of a visualization wand and steerableintroducer of a steerable introducer system, in accordance with certainembodiments of the disclosure.

FIG. 3A is a schematic view of an image frame associated with a firstintroducer orientation, in accordance with certain embodiments of thedisclosure.

FIG. 3B is a schematic view of an image frame associated with a secondintroducer orientation, in accordance with certain embodiments of thedisclosure.

FIG. 4 is a system schematic of a controller and introducer, inaccordance with certain embodiments of the disclosure.

FIG. 5 is a cut-away top view of a distal end of a steerable introducer,in accordance with certain embodiments of the disclosure.

FIG. 6A is a schematic view of an image frame associated with a firstintroducer orientation, in accordance with certain embodiments of thedisclosure.

FIG. 6B is a schematic view of an image frame associated with a secondintroducer orientation, in accordance with certain embodiments of thedisclosure.

FIG. 6C is a schematic view of an image frame associated with a thirdintroducer orientation, in accordance with certain embodiments of thedisclosure.

FIG. 7 is a flowchart of a method for steering an introducer, inaccordance with certain embodiments of the disclosure.

FIG. 8 is a flowchart of a method for steering an introducer, inaccordance with certain embodiments of the disclosure.

FIG. 9 is a flowchart of a method for adjusting introducer orientationto a frame of reference, in accordance with certain embodiments of thedisclosure.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

One or more specific embodiments of the present techniques will bedescribed below. According to an embodiment, a system is provided foraccessing patient anatomy with a steerable introducer, and for adjustingsteering commands according to an orientation of the introducer. As theintroducer is passed into a patient, the user may rotate or turn thedistal tip of the introducer in order to maneuver through the patient'sanatomy or to obtain a desired view. When the introducer is rotated orturned multiple times during a procedure, it can be difficult for theuser to keep track of the changed orientation of the introducer's distalend. Subsequently, the user may inadvertently bend or turn theintroducer in the wrong direction. For example, a user may intend tosteer the introducer to the user's right, but because the introducer isrotated from its default position, the result of this command is for theintroducer to bend to the user's left.

The disclosed embodiments use orientation information of the introducerto account for differences between the orientation of the distal end ofthe introducer and the user's own frame of reference. As a result, anintroducer steering system using the orientation information providesmore intuitive viewing of images captured by the introducer and/or moreintuitive steering of the distal end of the introducer within thehandle. Further, because the orientation information is not harvestedfrom a hand-held device that is manipulated by the operator, operatorvariability in the position or angle of the hand-held device during usewill not contribute to inaccurate orientation information.

Accordingly, in an embodiment, an introducer steering system translatessteering commands from the user's reference frame into the orientationof the introducer, to preserve the user's intention in steering theintroducer. An embodiment of a steerable introducer system is depictedin FIG. 1. The system includes a video laryngoscope 10 and a steerableintroducer 12. An introducer is a thin, elongated, flexible instrument(which may be relatively narrower, more flexible, and longer compared toa laryngoscope or an endotracheal tube) that can be inserted into ahandle cavity for exploration, imaging, biopsy, or other clinicaltreatments, including catheters, endoscopes (with a camera), blindbougies (without a camera), or other types of scopes or probes.Introducers may be positioned to extend into the airway and be steeredinto the airway passage (such as the pharynx, larynx, trachea, orbronchial tubes) by the user via advancement of the distal end to adesired position and, in certain embodiments, subsequent rotation orrepositioning of the introducer. Introducers may be tubular in shape.

The introducer 12 includes a proximal end 14 (nearest the user) and anopposite distal end 16 (nearest the patient), and in this example acamera 18 positioned at the distal end, for viewing the patient'sanatomy. The introducer 12 includes a distal steerable portion 20 whichcan bend, twist, turn, or rotate. The distal steerable portion 20 maymove within two dimensions (in a plane) or within three dimensions ofspace. The distal steerable portion 20 is steered by a steering system.The steering system may include one or more memory metal components(e.g., memory wire, Nitinol wire) that changes shape based on electricalinput, a piezoelectric actuators (such as the SQUIGGLE motor from NewScale Technologies, Victor N.Y.), a retractable sheath (retractable torelease a pre-formed curved component such as spring steel which regainsits curved shape when released from the sheath), mechanical controlwires, hydraulic actuators, servo motors, or other means for bending,rotating, or turning the distal end or components at the distal end ofthe introducer.

The proximal end 14 of the introducer 12 connects to a controller, whichmay be a re-usable or single-use disposable handle 22, or amulti-purpose medical device such as the video laryngoscope 10. Thevideo laryngoscope 10 includes a handle 30 with a proximal end 32 anddistal end 34. The handle 30 includes a display screen 36 mounted on aproximal side of a grip or handle 38.

The controller operates the steering system to steer the steerableportion 20 of the introducer, and includes a user input 24 to receivesteering commands from the user. As shown in FIG. 1, the user input 24may include buttons on the handle 22 or on the video laryngoscope 10.The user presses the buttons to indicate which direction to turn orsteer the introducer. The user input 24 may be located on the displayscreen 36, on the grip 38, or both. The user input 24 may be one or morephysical buttons (or switch, lever, joystick, or similar input),touch-sensitive graphics or icons on a touch screen (such as on thescreen 36), a keyboard, or other suitable user input.

As shown in FIG. 1, the video laryngoscope includes a camera stick 40extending from the distal end 34 of the handle 30. The camera stick 40includes an elongated arm 42 carrying a camera 44 at its distal end. Thecamera stick 40 fits inside a removable, disposable, transparent blade46. More information about laryngoscope blades can be found, forexample, in Applicant's U.S. Pat. Nos. 9,775,505 and 9,066,700. Imagesfrom the video laryngoscope camera 44 and/or from the introducer camera18 (if present) are displayed on the display screen 36.

In an embodiment, as shown in FIG. 1, the steerable introducer 12includes an orientation sensor 56 at the distal tip of the introducer.The orientation sensor 56 may be an inertial measurement unit (IMU),accelerometer, gyroscope, or other suitable sensor. The orientationsensor 56 is located inside the tubular housing of the introducer 12. Inan embodiment, the orientation sensor 56 is located very close to theterminus of the distal end 16 of the introducer, and may be co-locatedwith the camera 18 (if present), to enable the orientation sensor 56 tocapture much of the full range of movement of the distal end 16 and thecamera 18. In an embodiment, the orientation sensor 56 is placed at(e.g., positioned on or in) the distal end 16 of the steerable portion20, remote from the proximal end of the steerable portion 20, to placethe orientation sensor 56 away from the fulcrum of movement of thedistal end 16 and camera 18.

The disclosed embodiments that include the orientation sensor 56 at ornear the distal end 16 of the introducer 12 provide more accurateorientation information relative to implementations in which theorientation information is derived from an orientation sensor in thecontroller (such as the video laryngoscope, wand, or handle). In such anexample, information derived from a sensor located in the controllerrelies on an assumption that the orientation of the controller is thesame as the orientation of the distal tip. To maintain the conditionsfor that assumption, the user may be instructed to hold the controllerat a particular angle or position during operation. However, uservariability in controller positioning during operation may lead toinaccuracies in the reported orientation information. Accordingly,orientation information measured at a handheld device located proximallyof the introducer may not provide accurate information. Further,movement measured at the controller may not translate into correspondingmovement of the distal tip. For example, the handle of the introducermay have a degree of compliance, so rotation by the user at the proximalend is not perfectly transferred along the length of the introducer. Asanother example, along a tortuous path through a patient's anatomy,torsion and friction can create losses in rotation. In an embodimentdisclosed herein, the orientation sensor 56 positioned at or near thedistal end 16 of the introducer 12 provides more accurate orientationinformation than controller-based measurement of orientation.

As provided in the disclosed embodiments, accurate orientationinformation captured at or near the distal end of an introducer 12permits active image adjustment, providing more intuitive visualizationof introducer images and, in turn, more intuitive steering within anestablished frame of reference that can be oriented to gravity or to auser-defined frame of reference. Further, the introducer is steered atthe distal end 16 without physical rotation of the proximal end, ratherthan implementations in which distal rotation and orientation change isdriven by torsional force translated from the proximal end 14 to thedistal end 14. These introducer uses a steering system that is effectiveat the distal tip (such as push or pull wires) to bend the distal tip ina desired direction, even when the length of the introducer between theproximal and distal ends is slack; the introducer does not requiretorsional force to translate along the introducer housing from theproximal to the distal end. The introducer does not need to be straightor taught in order to translate steering inputs to the distal end.Distal bending and movement of the introducer is accomplishedindependent of the orientation, position, or movement of the proximalend of the introducer; steering is not physically coupled between theproximal end (such as the handle) and the distal end. Further, theintroducer system does not need to make any assumptions about how muchtorsional force was successfully translated (or lost) along the lengthfrom the proximal to distal end; rather, an orientation sensor at thedistal tip provides an orientation signal that indicates the currentorientation of the distal tip. In this manner, the structure of theintroducer 12 may be less torsionally stiff relative to implementationsin which the steering relies on torsional force transfer. Accordingly,in an embodiment the introducer 12 is an extruded structure with lowtorsional stiffness (low enough that torsional rotation does nottranslate from the proximal to the distal end). In an embodiment, theintroducer is a non-braided structure, such as an extruded polymer. Inan embodiment, the introducer is an extruded structure devoid oftorsional stiffeners such as braided wires or braided structures.

FIG. 2 shows another embodiment in which the controller is a wand 50,similar to the video laryngoscope 10 but without the camera stick 40.The wand 50 includes the user input 24 to receive steering commands fromthe user, and includes the display screen 36. As shown in FIGS. 1 and 2,the controller may take the form of a handle 24, video laryngoscope 10,or wand 50 with integrated display screen 36. The controller can alsotake the form of a separate (not integrated) touch screen display,located in the room (such as mounted on a cart or stand), spaced apartfrom the introducer. This touch screen communicates user inputs via awired or wireless connection to the introducer. In one embodiment, thehandle 24 is integrated with the tubular introducer 12, and the entiredevice is single use and disposable. In another embodiment, theintroducer is a two-part system, and the controller (handle, wand,laryngoscope, or other device) is removable from the introducer 12. Theintroducer 12 is then discarded after use, and the controller isretained and used again with a new tubular introducer. The controllerhouses power, display, steering control, and other functionality. Inthis manner, the endoscope introducer many be disposable while therelatively more costly and complex controller may be reused.

The introducer 12 can attach to the wand 50 from a top (proximal) end ofthe wand (such that the introducer extends up over the top of thescreen), or from a bottom (distal) end of the wand (such that theintroducer extends below away from the bottom of the screen). Theintroducer 12A in FIG. 2 is shown to indicate the option to connect theintroducer to the wand 50 from below the screen.

FIG. 3A and FIG. 3B depict a method of steering an introducer, includingtranslating steering commands from a user into executable actuatorcontrols within the orientation of the introducer. For example, in FIGS.3A-B, the introducer is a tubular endoscope 120 with a camera 118located at its distal end 116. The endoscope 120 also has a feature—suchas an orientation indicator, a working channel, a surgical tool, a lightsource, or other instrument—that is located at one angular positionaround the tubular endoscope. In FIG. 3A-B, this feature is anorientation marker 126, which is a visible indicia or marker thatindicates to the user which direction is up for the steering controls.The marker 126 can be formed by printed graphics, a groove or otherthree-dimensional feature, a glow-in-the-dark ink or indicia, or anactively powered light (such as a small LED strip or light). The marker126 is located on a top side of the endoscope 120, when the endoscope isin its default, resting position (not bent, twisted, or steered). Inimage A, the endoscope has been rotated 180 degrees from that position,such that the marker 126 is on the bottom of the endoscope.

A real-time image from the camera is shown on the display screen 136,which may be a display screen on a wand, a video laryngoscope, amonitor, or any other display screen in the medical facility. Imagesfrom the camera 118 may be transmitted through wired connections orwirelessly to the display screen 136. In FIG. 3A, the field of view ofthe endoscope camera includes an anatomical structure 152 inside apassage 154. In an example, the passage 154 is the trachea, and thestructure 152 is a tumor. In other cases, the passage 154 is agastrointestinal passage, a nasal canal, or any other anatomical lumen.The structure 152 can be a polyp, tumor, blood vessel, vocal cords,suture, stent, bifurcation of passages (such as bronchial passages, orthe carina), or any other visible anatomical or medical feature.

In FIG. 3A, the structure 152 appears toward the top of the displayscreen 136. The user may decide to steer the endoscope 120 toward thestructure 152, and give an “up” steering command (such as through a userinput 24). The user's steering command is based on the user's frame ofreference, such as the directions in the image on the display screen.However, in this situation, the user's intention in steering “up” is notthe same as the default resting “up” orientation of the endoscope. Theorientation of the endoscope 120 has been changed with respect to theuser's reference frame.

Accordingly, in an embodiment, the endoscope steering system translatesthe user's command into the endoscope's current orientation. In FIG. 3B,the user provides an “up” steering command which means to bend “up” inthe frame of reference of the display screen 136. The steering systemtranslates this for the endoscope such that the endoscope bends in adirection opposite the marker 126 (which is the “down” direction in theendoscope's default frame of reference). As shown in FIG. 3B, theendoscope bends toward the structure 152, and the structure 152 movesinto the center of the screen 136.

A schematic cut-away view of the distal end 16 of the introducer 12 isshown in FIG. 5. This figure shows the camera 18 positioned at theterminus 17 of the distal end 16 of the introducer 12, to obtain a clearview forward. The orientation sensor 56 is located just behind thecamera 18. In an embodiment, the orientation sensor 56 is adjacent thecamera 18. In an embodiment, the orientation sensor 56 is mounted on aflex circuit behind the camera 18. In an embodiment, the orientationsensor 56 is mounted on the same flex circuit as the camera 18, thoughthe orientation sensor 56 and the camera 18 need not be in communicationon the shared flex circuit. In an embodiment, the orientation sensor hasa size of between 1-2 mm in each dimension. It should be understoodthat, in certain embodiments, the introducer 12 is blind and there is nocamera 18 present.

The orientation sensor is an electronic component that senses theorientation or movement of the distal end of the introducer. Theorientation sensor contains a sensor or a combination of sensors toaccomplish this, such as accelerometers, magnetometers, and gyroscopes.The orientation sensor detects position and/or movement of the distaltip of the introducer and provides a signal indicating a change in theintroducer's orientation. An orientation sensor 156 is also illustratedin FIG. 3A-B, located at the distal end 116 of the introducer 120, justbehind the camera 118. In an embodiment, the signal from the orientationsensor is based on just the accelerometer (without utilizing othersensors such as a gyroscope or magnetometer). In an embodiment, anaccelerometer is used as the orientation sensor.

A schematic diagram of electrical components of a steerable introducersystem is shown in FIG. 4. In this embodiment, the system includes acontroller 210 (such as a video laryngoscope, handle, or wand) and anintroducer 212. The controller 210 includes a microprocessor 260, memory261, power source 262, display screen 236, user input 224, andassociated circuitry 263 (such as, for example, a wireless transceiverfor receiving and communicating data). When the controller is a videolaryngoscope, it also includes a camera and light source, among othercomponents. The introducer 212 includes a camera 218 (if present), alight source 264, an orientation sensor 256, and a steering system 265.

As depicted in FIG. 4, an orientation signal 266 is passed from theintroducer 212 (based on measurements from the orientation sensor 256)to the controller 210, and an actuation control signal 268 is passedfrom the controller 210 to the introducer 212. The orientation signalmay be produced by the orientation sensor located at a distal end of theintroducer. The orientation signal defines an angular orientation of thedistal end of the introducer with respect to gravity.

The orientation signal 266 and steering commands from the user input 224are sent to the processor 260, which translates the steering commandsinto the actuation control signal 268. The actuation control signal 268operates the steering system by including specific executableinstructions for the individual actuator(s) of the steering system 265on the introducer, to bend, twist, or move the steerable portion 20 ofthe introducer.

A method 700 for controlling a steerable introducer, according to anembodiment, is depicted in FIG. 7. The method includes receiving, froman orientation sensor, an introducer orientation signal (at block 701).For example, the signal can be the orientation signal 266 from FIG. 4,received from an IMU or accelerometer or other sensor. The introducerorientation signal defines an angular orientation of the distal end ofthe introducer. The method also includes receiving, from a user input, asteering command in a user reference frame (at block 702). The methodalso includes translating the steering command from the user referenceframe into the introducer orientation (at block 703). The method alsoincludes steering the introducer according to the translated steeringcommands (at block 704). These steps can be done by a processor (such asprocessor 260) located inside an introducer controller (such as alaryngoscope, wand, or handle).

The user reference frame is the frame in which the user is givingsteering directions. This reference frame could be aligned with thedirection of gravity (so that a steering command of “down” means downtoward the Earth). As another example, the reference frame could bealigned with an image on the display screen (so that a steering commandof “down” means down in the image). As another example, the referenceframe can be centered on a patient (so that a steering command of “down”means toward the patient's back, if the patient is lying on their side,or toward some other anatomical feature of the patient). These are justa few examples.

Another example method 800 is outlined in FIG. 8. In this example, themethod includes receiving a steering command and an orientation signal(at block 801). The method includes generating a variable actuationcontrol signal as a function of both the steering command and theorientation signal (at block 802). The method includes steering theintroducer according to the variable actuation control signal (at block803). This can be done, for example, by a processor that generates anactuator control signal with specific instructions to operate theactuator(s) of the steering system of the introducer, to move theintroducer in the direction specified by the user.

In this way, the actuation controls for the steering system are not tiedto the introducer's internal frame of reference. Instead, the steeringapplied to the introducer is variable with the introducer's orientation.The same steering command from a user's frame of reference (for example,“up” toward the top of a display screen) will be translated intodifferent actuator controls depending on how the introducer is oriented.Even with the same steering command from a user, the control signal thatis sent to the actuator(s) of the steering control system of theintroducer will vary with the introducer's orientation. For example,when the user inputs a command to bend “up” toward the top of thedisplay screen, the steering control system may bend the introducertoward the orientation marker (such as 326), or away from theorientation marker, depending on how the introducer is oriented. Thus,the control signal that operates the steering control system of theintroducer varies with the introducer's orientation as well as with theuser's steering commands.

In an embodiment, the steering system includes two, three, four, or moreactuators that control movement of the steerable tip of the introducer.In an embodiment, the steering actuation is accomplished by modeling thetip of the introducer as a circle, with the modeled actuators occupyingdiscrete locations about the circumference of the circle. At theselocations, the actuators act on the tip to bend or move the introducer.The circle is rotated according to the orientation signal from theorientation sensor, to indicate the orientation of the introducer withrespect to the user's defined reference frame. Thus, when a usersteering command is received (for example, bend “up” toward the top ofthe circle), the appropriate actions for each respective actuator can bedetermined. Each actuator is operated or energized proportionatelyaccording to its position on the circle with respect to the usercommand. It should be understood that the two or more actuators may belocated at any position in the introducer and that correlates to arespective modeled circumferential location.

In an embodiment, the user can define a custom reference frame, as shownfor example in FIGS. 6A-C, which illustrate a display screen 336 of avideo laryngoscope displaying two images, a first image 370 from acamera on a video laryngoscope (such as camera 44 from FIG. 1), and asecond image 372 from a camera on an endoscope 312 (such as camera 18from FIG. 1). As shown in FIG. 6A-C, the endoscope 312 is located withinthe field of view of the laryngoscope camera, so the endoscope 312 isvisible in the image 370. The endoscope includes an orientation marker326 visible on a surface of the introducer 312. The lower panel of FIG.6A is a schematic representation of a cross-section of the endoscope,with the orientation marker 326 shown at a top left position of theintroducer 312.

The patient's vocal cords 374 and trachea 376 are visible in the imageson the screen 336. However, the endoscope image 372 is rotatedcounter-clockwise, compared to the video laryngoscope image 370.Accordingly, a user may decide to manually rotate the endoscope totransition from the position in FIG. 6A into the position shown in FIG.6B. In FIG. 6B, the user has rotated the endoscope clockwise by an angle4). This rotation can be seen by the new position of the orientationmarker 326. After rotation, the endoscope image 372 is aligned with thevideo laryngoscope image 370. At this point, the user may enter acommand to establish the current orientation of the endoscope (in FIG.6B) as the desired reference orientation or frame of reference. This canbe done by pushing a button on the user input 24 or on a touch screen orother input. The controller then stores the endoscope's currentorientation at the time of the user input as the reference frame forfuture adjustments. Subsequently, when the user gives steering commands(such as up, down, turn, etc.), those commands will be interpreted inthis stored reference frame, and translated into movement of theendoscope based on the endoscope's orientation data. This enables theuser to decide what reference frame to use for steering commands. Forexample, steering can be oriented to the patient's handle, instead of togravity. While alignment with the laryngoscope image 370 is shown as anexample, the user could choose any other orientation to establish thereference frame.

After establishing the position in FIG. 6B as the desired referenceorientation, the system will correct steering and images to thatreference orientation. For example, in FIG. 6C, the user has furtherrotated away from the position shown in FIG. 6B such that the introduceris rotated clockwise by angle α. The introducer itself has rotated, asshown by the new position of the orientation marker 326 as seen in thelaryngoscope image 370. However, the second image 372 (from theintroducer) has not rotated. In FIG. 6C, the vocal cords and trachearemain upright, as they were oriented in FIG. 6B. The systemaccomplishes this by receiving information from the orientation sensorat the distal tip of the introducer, determining the amount of change(here, clockwise rotation by the amount of the angle α), and reversingthat movement to retain the image 372 in the same orientation as FIG.6B. Similarly, steering controls entered by the user in FIG. 6B or FIG.6C are interpreted according to the orientation of FIG. 6B, as describedabove. If the user instructs the introducer in FIG. 6C to steer “up”toward the top of the screen 336, the system will bend the introducer inthat direction, even though the orientation marker 326 is rotated awayfrom that position by the angle α.

In another embodiment, the reference frame can be established byautomatic image recognition. For example, returning to FIG. 6A-C, theprocessor on the controller may automatically recognize features in theimage, such as the vocal cords 376 in both images 370 and 372, based oncomputer vision techniques. These techniques may include, for example, asingle shot object detector (that can recognize anatomical structures),Haar feature-based cascade classifiers (to recognize anatomicalstructures), a neural net trained to output orientation based on knownanatomy, landmark alignment with an ensemble of regression trees, objecttracking once a useful feature is identified, or other computer visiontechniques. The processor can then establish a reference frame based onthe orientation of the vocal cords—for example, identifying “up” astoward the top of the vocal cords (such as toward the epiglottis 378).The processor can be programmed to recognize other anatomical structures(for example, the cross-sectional shape of the trachea, anterior vs.posterior positioning) and update or store the reference frame based onthose structures. Image recognition can help align the user's referenceframe with the patient anatomy, instead of with gravity.

In an embodiment, a user can transition from the dual-picture orpicture-in-picture display (as shown in FIGS. 6A-C) to an introduceronly (only image 372) display or laryngoscope only (only image 370) andvice versa. Based on the type of images or images displayed, thereference frame can be automatically adjusted. For example, alignment ofthe reference frame may be based on the orientation of the laryngoscope.Typically, the laryngoscope is positioned during use such that the imagecaptured by the laryngoscope camera is oriented to gravity, with the topof the image on the display screen generally being “up” relative togravity. However, certain procedures may involve different laryngoscopepositioning relative to the patient, such as in the case of the userfacing the patient and holding the laryngoscope rotated 180 degrees. Inthat case, the top of the laryngoscope image displayed on the displayscreen would actually correspond to a “down” direction relative togravity. To account for different positioning or alignment of thelaryngoscope relative to gravity, the alignment may be based onalignment to the laryngoscope image, which may or may not be aligned togravity. However, upon a change of display mode to introducer-onlydisplay, the reference frame can automatically switch to a gravity-basedalignment, which is determined by the orientation signal of theorientation sensor. Further, in an embodiment, the techniques may beused to establish a reference frame for steering commands when theintroducer is blind (e.g., blind bougie) and no camera image isdisplayed. Nonetheless, the steering commands can be translated to agravity-based or user-established reference frame and translated usingthe orientation signal information from the orientation sensor.

In FIG. 6B, the processor can also determine that the endoscope image372 is rotated with respect to the video laryngoscope image 370 by theangle θ. In an embodiment, the processor corrects the endoscope image372, rotating the image to align it with the video laryngoscope image370, even without rotating the actual endoscope. This step keeps the twoimages aligned so that the user can more easily view them at the sametime.

In an embodiment, the orientation signal 266 (FIG. 4) is used to adjustthe displayed endoscope image (such as image 372, or on any otherdisplay screen). The processor 260 may use the signal 266 toautomatically adjust the displayed image to a desired orientation, suchas adjusting the image to make sure that the upward direction (anterior,toward the patient's chest) remains upward (toward the top proximalsurface) on the display screen, even when the endoscope is rotated orturned inside the patient. As an example, the user may rotate theendoscope clockwise degrees (or any amount), as shown in FIG. 6C, suchas to better position the endoscope within the patient's anatomy. InFIG. 6C, the image on the display screen remains stationary, even whenthe endoscope is rotated. The orientation sensor 256 at the tip ordistal end of the endoscope registers the rotation, and themicroprocessor 260 rotates the image on the screen in the reversedirection (in this example, counter-clockwise) by the same amount. Ifthe endoscope is rotated again, in either direction, the microprocessoragain compensates, so that the image on the screen remains oriented withthe patient's anterior pointed upward on the display screen. In anotherembodiment, the microprocessor 260 receives realtime updated signalsfrom the orientation sensor 256 indicating the relationship between thedistal tip and gravity, so that the microprocessor can continuallyadjust the image to keep the direction of gravity pointed downward onthe laryngoscope display screen, even as the endoscope itself isrotated.

An example method 900 is outlined in FIG. 9 that may be used inconjunction with a picture-in-picture display or dual-picture display ofa multifunctional visualization instrument with steering control (e.g.,a video laryngoscope 10, see FIG. 1). In this example, the methodincludes displaying an image (e.g., an image 372, see FIG. 6A-C) from anendoscope camera of an endoscope on a display screen (block 902).Optionally, the method may also display a first video laryngoscope image(e.g., an image 370, see FIG. 6A-C) from a laryngoscope camera of avideo laryngoscope. A user can define a custom reference orientation orreference frame (block 904) via a user input or, alternatively, thesystem may automatically establish a reference frame based on gravity orimage processing. The orientation of the endoscope at the time of theuser input is established as the reference frame (block 906). That is,when using a user input to define the reference frame, the orientationof the endoscope at the time of user input is flagged or stored as thereference frame orientation. The orientation sensor subsequentlyprovides a current orientation signal that indicates that the endoscopedistal end, which includes the endoscope camera, has a differentorientation than the reference frame (block 908). For example, thecurrent orientation of the distal end may change as a result of usermanipulation or steering events to move (e.g., rotate) away from theorientation associated with the reference frame to a currentorientation. Accordingly, a subsequent or second endoscope imagecaptured at the current orientation is translated from the currentorientation (e.g., modified, rotated) to the reference frame (block910). In an embodiment, any received steering command (block 912)received at the updated orientated is translated from the updatedorientation to the reference frame (block 914) based on the amount anddirection of rotation to facilitate steering of the endoscope accordingto the translated steering command (block 916).

A user can also update the reference orientation throughout a procedure.For example, the steps outlined in FIG. 9 can be repeated to enable theuser to establish a new reference orientation. For example, if apatients shifts, is rotated, sits up or lies down, coughs, etc., theclinical user may decide to establish a new reference orientation forthe introducer, such that the system will rotate image information fromthe introducer to keep the images stationary in this referenceorientation and translate steering commands from the user to theintroducer. In an embodiment, the system establishes an automatic ordefault orientation (such as gravity down), and the user can override orchange this default orientation by establishing a new referenceorientation as outlined in FIG. 9.

An introducer with variable steering may be used to assist withendotracheal intubation. During endotracheal intubation, clinicians(such as an anesthesiologist or other medical professional) attempt tonavigate an endotracheal tube through a limited view through thepatient's mouth. Clinicians may rely on the relative position ofanatomical structures to navigate. During intubation, the arytenoidcartilage proves useful as an anatomical landmark; the vocal cords areanterior to the arytenoid cartilage, the esophagus posterior. In anembodiment of the present disclosure, the anterior direction is alignedwith the top of the user's display screen and set as the referenceorientation, so that anterior is maintained as “up” on the screen.During intubation, the user can input a command to steer an introducer“up” to pass the tip over the arytenoids and into the vocal cords. Then,the user can pass an endotracheal tube over the introducer and ensurethat the endotracheal tube passes into the trachea, rather than theesophagus. By contrast, if the user becomes disoriented andinadvertently steers the introducer into the esophagus (instead of thetrachea), esophageal intubation can result, causing seriouscomplications for the patient. Accordingly, a system in which the user'sorientation is maintained, and steering inputs are translatedaccordingly, can improve clinical practice.

While the present techniques are discussed in the context ofendotracheal intubation, it should be understood that the disclosedtechniques may also be useful in other types of airway management orclinical procedures. For example, the disclosed techniques may be usedin conjunction with secretion removal from an airway, arthroscopicsurgery, bronchial visualization (bronchoscopy), tube exchange, lungbiopsy, nasal or nasotracheal intubation, etc. In certain embodiments,the disclosed multifunctional visualization instruments may be used forvisualization of anatomy (stomach, esophagus, upper and lower airway,ear-nose-throat, vocal cords), or biopsy of tumors, masses or tissues.The disclosed multifunctional visualization instruments may also be usedfor or in conjunction with suctioning, drug delivery, ablation, or othertreatments of visualized tissue. The disclosed multifunctionalvisualization instruments may also be used in conjunction withendoscopes, bougies, introducers, scopes, or probes.

In operation, a caregiver may use a laryngoscope to assist inintubation, e.g., to visualize a patient's airway to guide advancementof the distal tip of an endotracheal tube through the patient's oralcavity, through the vocal cords, into the tracheal passage.Visualization of the patient's anatomy during intubation can help themedical caregiver to avoid damaging or irritating the patient's oral andtracheal tissue, and avoid passing the endotracheal tube into theesophagus instead of the trachea. The laryngoscope may be operated witha single hand (such as the user's left hand) while the other hand (suchas the right hand) grips the endotracheal tube and guides it forwardinto the patient's airway. The user can view advancement of theendotracheal tube on the display screen in order to guide theendotracheal tube into its proper position.

While the video laryngoscope can facilitate more efficient intubationthan direct-view intubation, certain patients may benefit fromvisualization and/or steering devices that extend further into theairway than a laryngoscope. For example, patients with smoke inhalation,burns, lung cancer, and/or airway traumas may benefit from visualizationpast the vocal cords, which is not accomplished with a laryngoscope.Such visualization may be beneficial for endoscopic placement ofendotracheal tubes and/or placement or positioning of suctioning devicesin the airway. Endoscope placement (e.g., with an endotracheal tubeloaded into the endoscope) may be helpful for anterior or challengingairways. For example, patients whose anatomy cannot be suitablymanipulated (either through head positioning or laryngoscopy) to createspace for passage of an endotracheal tube may benefit from imagingdevices that go beyond the visualization range of a laryngoscope andthat provide a greater steering range for a camera, or from articulatingdevices that can be manipulated and moved within the visualization rangeof the laryngoscope.

While the disclosure may be susceptible to various modifications andalternative forms, specific embodiments have been shown by way ofexample in the drawings and have been described in detail herein.However, it should be understood that the embodiments provided hereinare not intended to be limited to the particular forms disclosed.Rather, the various embodiments may cover all modifications,equivalents, and alternatives falling within the spirit and scope of thedisclosure as defined by the following appended claims.

1. A steerable introducer system, comprising: a laryngoscope comprising:a handle comprising a proximal end and a distal end; a display screen onthe handle; a laryngoscope camera at the distal end of the handle; aprocessor within the laryngoscope; and a steering input located on thehandle or the display screen; and an introducer coupled to the handleand comprising an orientation sensor at a distal end of the introducer;wherein the processor within the laryngoscope is programmed to executeinstructions for: receiving from the steering input a steering commandin a first reference frame, and mapping the steering command from thefirst reference frame to a second reference frame oriented to the distalend of the introducer based on an orientation signal from theorientation sensor.
 2. The system of claim 1, wherein the processor isfurther programmed to execute instructions for generating a controlsignal for steering the introducer according to the mapped steeringcommand.
 3. The system of claim 1, wherein the second reference frame isdefined by an angular offset from the first reference frame.
 4. Thesystem of claim 3, wherein mapping the steering command to the secondreference frame comprises adjusting the steering command by the angularoffset.
 5. The system of claim 1, wherein the first reference frame isdefined by a user input.
 6. The system of claim 1, wherein the firstreference frame is defined by automatic image recognition.
 7. The systemof claim 6, wherein the processor is programmed to receive an image fromthe laryngoscope camera to identify a feature of the image to performthe automatic image recognition.
 8. The system of claim 6, wherein theprocessor is programmed to receive an image from an introducer camera atthe distal end of the introducer to identify a feature of the image toperform the automatic image recognition.
 9. The system of claim 1,wherein the display screen displays an image from the laryngoscopecamera in the first reference frame.
 10. The system of claim 9, whereinthe display screen displays an image from an introducer camera at thedistal end of the introducer in the first reference frame.
 11. Anendoscope controller, comprising: a handle comprising a proximal end anda distal end; a display screen on the handle; an endoscope port locatedon the handle or the display screen; a user input located on the handleor the display screen; and a processor within the controller, programmedto execute instructions for: receiving, from the user input, a steeringcommand in a user reference frame, receiving, from an endoscope coupledto the endoscope port, an orientation signal from an orientation sensorat an endoscope distal end, and translating the steering command as afunction of the orientation signal.
 12. The controller of claim 11,wherein the processor is further programmed to execute instructions forsteering the endoscope according to the translated steering command. 13.The controller of claim 11, further comprising the endoscope coupled tothe endoscope port, wherein the endoscope comprises an orientationsensor that generates the orientation signal.
 14. A method forcontrolling a steerable introducer, comprising: receiving, at aprocessor, an orientation signal from an orientation sensor located at adistal end of a steerable introducer, the orientation signal defining anangular orientation of the distal end of the introducer; receiving, atthe processor, a steering command comprising a steering direction in auser reference frame; translating the steering command from the userreference frame to the angular orientation of the distal end of theintroducer; and steering the distal end of the introducer according tothe translated steering command.
 15. The method of claim 14, wherein theuser reference frame is defined in reference to an anatomical feature ofthe patient.
 16. The method of claim 14, wherein the user referenceframe is defined by a user input.
 17. The method of claim 14, furthercomprising: receiving, at the processor, an image from a camera at thedistal end of the introducer; rotating the image into the user referenceframe; and displaying the rotated image at a display screen. 18.-20.(canceled)